Add external OpenSearch support with E2E test#268
Closed
Conversation
Allow photon-docker to connect to a self-hosted OpenSearch instance instead of relying on photon's embedded one. When OPENSEARCH_TRANSPORT_ADDRESSES is set, the container skips index downloads, passes -transport-addresses and -cluster to photon.jar, and forces UPDATE_STRATEGY=DISABLED. - config.py: add OPENSEARCH_TRANSPORT_ADDRESSES and OPENSEARCH_CLUSTER env vars - entrypoint.py: skip download logic when external OpenSearch is configured - process_manager.py: conditional command construction (external vs embedded), skip initial setup and lock file cleanup for external mode - validate_config.py: warn when REGION is set alongside external OpenSearch - docker-compose.external-opensearch.yml: user-facing compose for external setup - external-opensearch-test.yml: E2E GitHub Actions workflow that downloads andorra index, pre-loads it into standalone OpenSearch 3.x via dangling index import, and verifies photon health check passes https://claude.ai/code/session_01LJJEVsnKGopysw7LtMHBQi
The download server organizes sub-regions under their continent: /europe/andorra/photon-db-andorra-1.0-latest.tar.bz2 https://claude.ai/code/session_01LJJEVsnKGopysw7LtMHBQi
…ta mounting The previous approach downloaded a pre-built index tar.bz2 and tried to mount it into OpenSearch 3.0.0 via dangling index auto-import. This failed because: - OpenSearch 3.0.0 dropped the `es.` system property prefix (replaced by `opensearch.`), so `-Des.gateway.auto_import_dangling_indices=true` was silently ignored - The embedded OpenSearch data format from photon is not directly compatible with standalone OpenSearch 3.0.0 Replace the broken data-mounting strategy with photon.jar's own import command using a minimal JSONL dump file. This creates the index with the correct mapping and is version-independent. Also fix the docker-compose comments to use the correct `-import-file` parameter (was incorrectly documented as `-json-dump`). https://claude.ai/code/session_01LJJEVsnKGopysw7LtMHBQi
OpenSearch 3.0.0 dropped the `es.` system property prefix. The dangling index auto-import setting was silently ignored because `-Des.gateway...` is not recognized — it must be `-Dopensearch.gateway...`. https://claude.ai/code/session_01LJJEVsnKGopysw7LtMHBQi
Owner
Author
|
lmao |
OpenSearch 3.0.0 ignores cluster settings passed via -D JVM system properties. Move gateway.auto_import_dangling_indices to opensearch.yml where it is actually read. Also set cluster.name to "photon" so it matches the OPENSEARCH_CLUSTER value photon uses to connect. https://claude.ai/code/session_01LJJEVsnKGopysw7LtMHBQi
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Allow photon-docker to connect to a self-hosted OpenSearch instance instead of relying on photon's embedded one. When OPENSEARCH_TRANSPORT_ADDRESSES is set, the container skips index downloads, passes -transport-addresses and -cluster to photon.jar, and forces UPDATE_STRATEGY=DISABLED.